Using Fault Injection to Analyze the Scope of Error Propagation in Linux

Operating systems (OSes) are crucial for achieving high availability of computer systems. Even if applications running on an operating system are highly available, a bug inside the kernel may result in a failure of the entire software stack. The objective of this study is to gain some insight into the development of the Linux kernel that is more resilient against software faults. In particular, this paper investigates the scope of error propagation. The propagation scope is process-local if the erroneous value is not propagated outside the process context that activated it. The scope is kernel-global if the erroneous value is propagated outside the process context that activated it. The investigation of the scope of error propagation gives us some insight into 1) defensive coding style, 2) reboot-less rejuvenation, and 3) general recovery mechanisms of the Linux kernel. For example, if most errors are process-local, we can rejuvenate the kernel without reboots because the kernel can be recovered simply by killing faulty processes. To investigate the scope of error propagation, we conduct an experimental campaign of fault injection on Linux 2.6.18, using a kernel-level fault injector widely used in the OS community. Our findings are (1) our target kernel (Linux 2.6.18) is coded defensively. This defensive coding style contributes to lower rates of error manifestation and kernel-global errors, (2) the scope of error propagation is mostly process-local in Linux, and (3) global propagation occurs with low probability. Even if an error corrupts a global data structure, other processes merely access to them.